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The Longitudinal Emergency Medical Technician Attributes and Demographics Study (LEADS) provides 
a representative sampling of EMTs throughout the United States. This study examines the relationship 
between instructor quality and National Registry of Emergency Medical Technicians certification 
examination outcomes. Results show significant correlations, based on instructor quality on initial exam 
scores as well as on the number of tries to obtain a passing score. Implications for HRD practitioners, 
adult educators, and researchers are discussed. 
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Emergency medical technicians (EMTs) represent an important segment of the healthcare system by providing pre- 
hospital care to those who are critically ill or injured. “Paramedics are critical members of the emergency medical 
services system that treat and transport approximately 25 to 30 million patients per year” (National Highway Traffic 
Safety Administration, 2001). This represents a significant portion of the almost 117 million annual emergency 
room visits in 2004 (American College of Emergency Physicians, 2005). A critical factor in ensuring high-level care 
involves initial and ongoing training and certification (Dawson, Brown, & Harwell, 2003). 

Given the importance of EMT education and training, it is surprising that only a few studies have examined 
issues related to that training. Janing (2001), Stolk, Alexandrian, Gros, and Paggio (2001), Janing and Sime (2001), 
Jones and Cookson (2002), and DeLorenzo and Abbott (2004) investigated various aspects of training. Cannon, 
Mennagazzi, and Margolis (1999) determined the effects or transfer of training on pediatric care, and Russ-Eft, 
Dickison, and Levine (2005) studied the effects of instructor quality and instructional materials quality on transfer of 
training as measured by self-ratings of preparedness. The present study continues research into issues related to 
transfer of EMT training. 

Theoretical Framework 

Transfer of training describes the process by which knowledge, skills, and abilities learned during training are 
transferred or displayed in other settings. Baldwin and Ford (1988) suggested a framework that included training 
inputs, training outputs, and transfer conditions; Broad and Newstrom (1996) focused on transfer elements before, 
during, and after training; and Russ-Eft (2002) identified situational, pre-training, training design, and post-training 
elements. Recent work on transfer, however, has identified the importance of the transfer or organizational 
environment (Hawley & Barnard, 2005; Holton, Bates, Seyler, & Carvalho., 1997; Holton, Bates, & Ruona, 2000; 
Rouillier & Goldstein, 1993). Although the organizational (or transfer) climate affects the degree of transfer, 
educational, training, HRD practitioners have focused much of their work on the design and implementation of the 
training itself. So, questions arise as to the effects of these design and instructional methods. 

Lim and Morris (2006) identified two major training design factors: content design and instructional methods. 
Content design refers to such aspects as the degree of similarity between the training content and the transfer task 
(e.g., Axtell, Maitlis, & Yearta, 1997; Kontoghiorghes, 2002). Instructional methods include such aspects as spaced 
practice, overlearning, and goal setting (e.g., Dempster, 1988; Driskell, Willis, & Cooper, 1992; Wexley & 
Nemeroff, 1975; Werner et. al, 1994). Indeed, Taylor, Russ-Eft, and Chan (2005), in a meta-analytic study of 
behavior modeling training, showed the effects of both content design and instructional methods. 
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But, all of this research fails to address the role of the instructor in training. Recently, however, Russ-Eft, 
Dickison, and Levine (2005) showed the effects of instructor quality and instructional materials quality on self- 
ratings of EMT preparedness. These researchers indicated that a major problem with that study revolved around the 
issue of common-method bias - that is both ratings of quality and preparedness came from the EMTs themselves. To 
overcome that problem, a study is needed that uses separate measures and separate sources. 

Research Problem 

The National Registry of Emergency Medical Technicians (NREMT) and the U.S. Department of Transportation, 
National Highway Traffic Safety Administration (USDOT/NHTSA) have identified important issues related to the 
work of EMTs, including the quality of the education and training being received. The NHTSA has been designated 
as the agency within the federal government to work with the State EMS lead agencies to create national 
standardized education programs for EMTs. These two agencies have collaborated on issues related to EMT 
education and training (e.g.. Brown et ah, 1999). Together the two agencies identified the need to research the EMS 
workforce as an aid to better understand the impact of future decisions from a regulatory and educational 
perspective. 

The present study focused on issues related to the education and training of EMTs as affecting the outcomes 
experienced on the NREMT certification examination received by EMTs. It represents an extension of previous 
work by Russ-Eft, Dickison, and Levine (2005) that examined the effects of instructor and materials quality on EMT 
preparedness. The major questions in the present study focused on whether there is a relationship of the quality of 
the instructor and of the instructional materials with later performance on the NREMT certification examination. 

HI - There will be no correlation between quality of instructors (or at least perceived quality by the EMTs) and 
initial certification exam scores. 

H2 - There will be no correlation between quality of instructors and number of test attempts required by EMTs to 
pass the exam. 

H3 - There will be no correlation between quality of materials and initial certification exam scores. 

H4 - There will be no correlation between quality of materials and number of test attempts required to pass exam. 

Methods 

Background 

This Longitudinal EMT Attributes and Demographics Study (LEADS) is a joint venture between NREMT and 
NHTSA. The project is lead by a team of researchers who have experience as State EMS Directors, State EMS 
Training Coordinators, EMS System Managers, Emergency Physicians, EMS Educators, and survey researchers. It 
also includes staff of the NREMT and NHTSA. More about the LEADS effort can be found in Brown, Dickison, 
Misselbeck, and Levine (2002) and in Russ-Eft, Dickison, and Levine (2005). 

Sample 

The sample for the LEADS studies, including the present study, comes from the NREMT database. At the time 
of the study, the NREMT database included over 110,000 EMTs who were nationally registered in the U.S. The 
sampling frame for this study involves two levels: EMT-Basics and EMT-Paramedics; also, these individuals were 
registered at those respective levels by the NREMT, as of 15 September 1999. Separate sampling frames were 
developed for the two levels, and these frames were stratified by duration of continuous registration at each level 
(“new”, referring to those registered at the level for less than one year versus “old”, referring to those registered for 
one year or longer) and by race (“white”, referring to those who self-identified as white, other, or did not self- 
identify versus “minority”, referring to those who self-identified as Asian, Black, Hispanic, or Native American.). 
Sample sizes were intended to maximize the efficiency of the sample for comparing different types of EMTs as well 
as for producing estimates of population parameters. The sizes of the strata and return rates appear in Table 1 . 

Case weights were calculated for respondents in each stratum, reflecting the individual’s probability of 
selection. These case weights were adjusted, within strata, for non-response. 

LEADS Instrument Development. 

The survey instrument included two sections. The first section, the “core,” contained 78 items and, given the 
longitudinal design of the study, will continue to be readministered in each annual survey. The second section, the 
“snapshot,” has a different focus in each mailing. The snapshot survey that was used for this study was 




Table 1 : Frames and Sample Characteristics 





Frame Size 


Sample Size 


Number of 
Respondents 


Response Rate 


EMT-Basics 


White, New 


23,432 


1,050 


354 


33.7% 


White, Old 


45,804 


1,050 


296 


28.2% 


Minority, New 


2,093 


551 


129 


23.4% 


Minority, Old 


2,799 


551 


101 


18.3% 


Total 


74,128 


3,202 


880 


27.5% 




EMT-Paramedics 


White, New 


6,376 


840 


302 


36.0% 


White, Old 


27,828 


840 


341 


40.6% 


Minority, New 


646 


442 


137 


31.0% 


Minority, Old 


1,075 


440 


130 


29.5% 


Total 


35,925 


2,562 


910 


35.5% 



comprised of 41 items, focusing on EMS education and training. Copies of the survey are available on-line 
(http://www.nremt.org/about/lead_survey.asp). 

Draft survey instruments were reviewed, modified, and pilot-tested. For the survey used in this study, 42 EMT- 
Basics and EMT-Paramedics at 8 different locations received the pilot-test instrument. Based on the pilot test, 
additional response categories were developed for several items, several items underwent minor revisions, and 
several items were deleted. 

LEADS Data Collection Procedures 

Surveys were mailed out on September 17, 1999 to the selected EMT-Basics and EMT-Paramedics. These were 
sent with a postage-paid return envelope and a letter describing the goals of the project and assuring confidentiality. 
Two round trip airline tickets to any location within the continental United States were offered to one randomly 
selected participant as an incentive to respond. Returned surveys were scanned by an optical reader and used to 
create an analytic data file. 

LEADS Non-Respondent Survey 

An abbreviated version of the survey, containing demographic, attitudinal, income, and educational items from 
the regular survey, was mailed out to 500 EMT-Basics and 500 EMT-Paramedics survey non-respondents in March 
2000. These “non-respondent” surveys were received from 154 EMT-Basics (30.8 percent) and 207 EMT- 
Paramedics (41.4 percent), optically scanned, and used to create analytic data files. Responses from EMT-Basics 
and EMT-Paramedics who completed and returned the non-respondent survey were compared with the responses of 
demographically similar regular survey respondents. 

Discriminant analysis indicated differences in the response patterns of three types of EMT-Paramedics 
respondents and non-respondents (minority, new; minority old; white, new), comprising less than a quarter (22.5 
percent) of the total number of EMT-Paramedics. These differences appeared in responses to an item about EMS 
income in the past 12 months: Non-respondents reported significantly higher incomes than respondents. When this 
earning item was excluded, there was little evidence that non-respondents differed from respondents on 
demographic, attitudinal, or education items (Brown, Dickison, Misselbeck, & Levine, 2002). 

Measures Used in the Present Study 

Data for the present study came from two separate sources. The first source involved the LEADS survey 
instrument as described above. The second source was the records concerning the NREMT certification 
examinations. The resulting measures will be described in the following paragraphs. 

LEADS analysis items and categorizations. The items examined in the current study focused on the quality of 
the instructors, the quality of the materials, and the EMT’s self-assessed level of preparedness for performing 
various EMT tasks. Items on instructor quality asked about (1) technical knowledge, (2) practical knowledge, (3) 
teaching ability, (4) enthusiasm, (5) availability outside of class, and (6) professionalism. EMTs were asked to 
respond on a scale of “excellent” = 4, “good” = 3, “fair” = 2, and “poor” =1. In addition, the mean scores on each of 
the instructor quality items were combined to create a variable called “total instructor quality.” Items on the quality 
of the materials asked about (1) the textbook, (2) audiovisual materials (videos, slides), and (3) course equipment 
(mannequins, splints, etc.). These three items used the same rating scale as that used for rating instructor quality. 





NREMT certification examination test scores. The NREMT written examination consists of items drawn from 
the National Registry’s item bank following a test plan. This test plan is based upon the results of the EMT-Basic 
and EMT-Paramedic Practice Analysis conducted in 1995 and 1999. This Practice Analysis involved surveys of 
hundreds of EMTs at the respective levels who were asked about: (1) the frequency they performed and (2) the 
importance of various tasks, including assessment and patient management, operational, and intervention tasks. 

Test items are prepared by Item Writing Committees consisting of 10 to 20 EMS experts. These items are then 
pilot tested throughout the United States. Following the pilot test, a Standard Setting and Final Review Committee, 
comprised of providers in pre-hospital care and varying in credentials from EMT-Basics, Intermediates, Paramedics, 
Nurse-Paramedics, State Directors, State Training Coordinators, and Physicians, determines the pass/fail score using 
a modified Nedelsky formula (Nedelsky, 1954). Using the results from this Committee, examinations are developed 
following the test plan and have a pass-fail score of 70% to pass. Also, the EMT-Paramedic exam includes 
minimum scores for each part. 

It should also be noted the EMT written examination is a criterion-based rather than a norm-based exam. “The 
purpose of the NREMT test is to identify entry-level competent EMTs that are able to safely and effectively 
practice” (National Registry of Emergency Medical Technicians, n.d., T| 19). Thus, candidates taking the exam are 
not in competition with others but must simply “demonstrate they have enough knowledge so that they can safely 
and effectively practice” (T| 21). Furthermore, because this is a criterion-based exam, test scores tend to bunch 
around the cut-off and to show a narrow range. Another characteristic of criterion-based exams is that they do not 
rank individuals, as do norm-based exams, rather they identify individuals who have met the criteria. 

In the present study two separate measures were used. The first measure involved the actual test score from the 
first attempt by each candidate. For EMT-Basics, the exam contains 150 items, while for EMT-Paramedics, it 
contains 180 items. 

Analysis Procedures 

Identifying information was used by NREMT staff to develop a data file containing NREMT certification 
examination scores for LEADS participants. The LEADS data files were then matched with the NREMT 
certification examination data file. The exam data file contained two types of information: (1) exam scores for each 
attempt until passing the exam, and (2) the total number of attempts to pass the exam. Descriptive statistics were 
obtained for the certification exam data and then correlations of the certification exam data with the ratings on 
instructor quality and instructional materials quality were run separately for EMT-Basics and EMT-Paramedics. 

Results 

Table 2 presents examination score results from the most recent first attempt, using the weighted data. Because a 
complex sampling plan, with respondents in different strata having different probabilities of selection, was used in 
the LEADS Survey, the statistical significance of the correlation coefficients was not determined using the actual 
sample sizes. Such a procedure would have over-estimated the level of significance. Instead, the effective sample 
sizes were estimated, using Kish’s method (i.e., (sum of the weights)^/sum of each weight^.). This procedure 
calculates the size of a simple random sample that would have equivalent precision to the stratified, 
disproportionately selected sample used in LEADS. Further details can be found at: 
www.statistics.gov.uk/downloads/theme_otherAVeighting_Subject.pdf 

Effective sample sizes were calculated for EMT-Paramedics; EMT-Basics, and a combined universe of EMT- 
Basics and EMT-Paramedics and used to determine the statistical significance of the correlation coefficient. 



Table 2: NREMT Certification Examination Scores for the Weighted Data 



Level 


Effective 
Sample Size 


Mean 


Std. Dev. 


Minimum 


Maximum 


Basics 


513 


116.02 


3.19 


78 


141 


Paramedics 


406 


127.05 


3.94 


67 


165 


Total 


855* 


119.38 


3.37 


67 


165 



* NOTE: One cannot add the effective sample sizes for Basics and for Paramedics to calculate the effective sample 
size for the combined universe. 



Table 3 presents the weighted frequencies indicating the number of attempts taken to pass the exam. Table 4 
presents the weighted proportions of number of attempts. Note that after six attempts, a candidate must complete 
another certification course. 





Table 3: Weighted Frequencies of Number of Attempts to Pass the NREMT Certification Examination 



Level 


1 Attempt 


2 Attempts 


3 Attempts 


4 Attempts 


5 Attempts 


6 Attempts 


Basic 


58,310 


2,694 


578 


458 


29 


0 


Paramedic 


23,783 


2,041 


595 


565 


174 


83 


Total 


82,093 


4,735 


1,173 


1,023 


203 


83 



Table 4: Weighted Proportions of Number of Attempts to Pass the NREMT Certification Examination 



Level 


1 Attempt 


2 Attempts 


3 Attempts 


4 Attempts 


5 Attempts 


6 Attempts 


Basic 


93.9% 


4.3% 


0.9% 


0.7% 


0.1% 


0.0% 


Paramedic 


87.3% 


7.5% 


2.2% 


2.1% 


0.6% 


0.3% 


Total 


91.9% 


5.3% 


1.3% 


1.2% 


0.2% 


0.1% 



Table 5 presents the results for EMT-Basics concerning correlations of various dimensions of instructor quality and 
materials quality with (1) certification examination scores from the first attempt and (2) the total number of 
attempts. None of the EMT-Basics ratings of instructor quality or of instructional materials quality appear to be 
significantly correlated with certification examination scores. There are, however, significant correlations with the 
number of attempts, showing that certain characteristics of the instructor (specifically, practical knowledge, 
enthusiasm, and availability outside of class) correlate with fewer attempts required to pass the certification exam. 



Table 5 : Correlations for Instructor and Materials Quality with Certification Exam Scores and Number of 
Certification Exam Attempts for EMT-Basics 



Item 


Exam Scores 


Number of Attempts 


Instructor: Technical knowledge 


.079 


-.086 


Instructor: Practical knowledge 


.077 


-.089 (p < .05) 


Instructor: Teaching ability 


-.002 


-.055 


Instructor: Enthusiasm 


.077 


-.103 (p< .05) 


Instructor: Availability outside class 


.029 


-.094 (p < .05) 


Instructor: Professionalism 


.008 


-.074 


Mean of Instructor Quality 


.042 


-.089 (p < .05) 


Materials: Textbook 


-.010 


-.025 


Materials: Audiovisual materials 


-.034 


-.003 


Materials: Course equipment 


-.053 


.026 


Mean of Instructional Materials 


-.041 


.003 



Table 6 presents the results for EMT Paramedics concerning correlations of various dimensions of instructor quality 
and materials quality with (1) certification examination scores from the most recent first attempt and (2) the total 
number of attempts. These results show that certain ratings of instructor quality (specifically, practical knowledge 
and enthusiasm) are significantly correlated with both exam scores and with a smaller number of attempts needed to 
pass. 



Table 6: Correlations for Instructor and Materials Quality with Certification Exam Scores and Number of 
Certification Exam Attempts for EMT-Paramedics 



Item 


Exam Scores 


Number of Attempts 


Instructor: Technical knowledge 


.115 (p< .05) 


-.049 


Instructor: Practical knowledge 


.133 (p< .01) 


-.125 (p< .05) 


Instructor: Teaching ability 


.025 


-.091 


Instructor: Enthusiasm 


.103 (p< .05) 


-.125 (p< .05) 


Instructor: Availability outside class 


.073 


-.063 


Instructor: Professionalism 


.036 


-.037 


Mean of Instructor Quality 


.083 


-.048 


Materials: Textbook 


-.018 


.024 


Materials: Audiovisual materials 


.061 


-.038 


Materials: Course equipment 


-.013 


-.020 


Mean of Instructional Materials 


-.019 


-.001 






Discussion and Implications 



The results of the present study yield important findings related to EMT training in particular and to the literature on 
transfer of training in general. The findings suggest that the quality of the instructional materials is less important 
than the quality of the instructor in terms of training outcomes. Furthermore, certain instructor characteristics, such 
as practical knowledge and enthusiasm, appear to be of greater importance than other instructor characteristics, such 
as professionalism. These findings provide support to the earlier work by Russ-Eft, Dickison, and Levine (2005) 
indicating the effects of instructor quality on EMT preparedness. 

It is important to comment on two possible limitations of these correlational findings. First, correlations indicate 
that the variables are related but not necessarily that one causes the other (Gall, Gall, & Borg, 2003, pp. 323-324). 
Such correlational results are common in HRD literature, given the popularity of regression analyses and structural 
equation modeling (e.g., Naquin & Holton, 2002). To overcome this limitation, one would need to conduct a 
controlled experiment, probably undertaken in a laboratory setting. The second limitation related to these 
correlations involves the size of the correlations, which though significant, may appear to be small. The 
Psychological Assessment Work Group of the American Psychological Association (Meyer, et. ah, 2001), however, 
in a meta-analysis of correlations involving medical and psychological tests indicated that, except for reliability 
coefficients (typically r = .70 or greater) or monomethod validity coefficients (typically r = .5 or greater), it is 
challenging and unusual to obtain “univariate correlations that are much above .30” (p. 133). Thus, for example, 
they presented such correlations: r = .02 for aspirin and reduced risk of death by heart attack, r = .08 for calcium 
intake and bone mass in premenopausal women, r = .1 1 for antihistamine use and reduced runny nose and sneezing, 
and r = .18 for nicotine patch [versus placebo] and smoking abstinence at end of treatment and at six-month follow- 
up (see pp. 130 to 132 for a complete listing).. 

A final limitation of the study is that it only includes analyses of available data. Given the lack of survey 
information concerning the transfer environment, we cannot compare the relative strength of instructor quality as 
compared with the environmental factors (e.g., Hawley & Barnard, 2005; Holton et al, 1997, 2000). However, since 
candidates cannot function within a work environment as an EMT prior to being certified, it is unlikely that such 
environmental characteristics would have much impact. 

Such results can be used by HRD practitioners and those involved in EMT education and training to identify the 
characteristics needed for successful training. For example, “availability outside of class” showed a significant 
negative correlation with the number of attempts to pass, but only for EMT Basics. In contrast, “technical 
knowledge” showed a significant positive correlation with exam scores, but only for paramedics. For both groups, 
however, ratings of “practical knowledge” and “enthusiasm” were significantly correlated with later outcomes. Such 
results appear to corroborate Galbraith’s (1998) comment that “being technically proficient is not enough. The 
teacher of adults must also possess personality characteristics, interpersonal skills, and positive behaviors” (pp. 4-5). 

These results also contribute to research on transfer of training, specifically that of training reactions and ratings 
of training. Although Kirkpatrick (1994) suggested that positive reactions contribute to learning and performance 
transfer, Bretz and Thompsett (1992) and Dixon (1990) found no such relationships. Alliger et al. (1997) meta- 
analytic study, however, showed a positive relationship between utility ratings and performance, while no such 
relationship existed with affective ratings. The present study may provide further details. Specifically, it shows that 
ratings related to utility - those of practical knowledge - correlated with certification exam outcomes. 

The positive correlations of ratings of instructor enthusiasm represent an area that may need further exploration. 
It may be that such enthusiasm contributes to trainee motivation to learn or motivation to transfer. On the other 
hand, it may be that instructor enthusiasm for EMT candidates may represent a form of supervisor or peer support 
and mentoring. Further studies of this particular instructor characteristic and its relationship to transfer of training - 
for EMTs and for other trainees are warranted. 
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